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ABSTRACT: This paper describes theory-led design as a way of developing novel tools for learning 
analytics (LA). It focuses upon the domain of automated discourse analysis (ADA) of group 
learning activities to help an instructor to orchestrate online groups in real-time. The paper 
outlines the literature on the development of LA tools within the domain of ADA, and poses an 
argument for conducting tool development based upon first-principles. It describes first 
principles as being drawn from theory and that these principles can subsequently inform the 
structure and behaviour of tools. It presents a framework for this process of theory-led design. 

The framework is substantiated through the example of developing a new tool for assisting 
instructors with the orchestration of online groups. A description of the tool is given and 
examples of results from use with real-world data are presented. The paper concludes with a call 
for intent on the part of designers to connect the design process explicitly to theory on the basis 
that this has the potential to yield innovation when developing tools as well as the prospect of 
outcomes from tools connecting back to theory. 

Keywords: Automated discourse analysis, collaborative learning, orchestration, learning 
analytics, topic detection 

Editor's Note: As part of the Special Section on Learning Analytics & Learning Theory this article is followed by a 
short commentary on pp. 44-46 that discusses the challenges it faced and successes it achieved in drawing on and 
contributing to theory use in learning analytics. 

1 INTRODUCTION 

Learning analytics (LA) has in recent years moved from early successes in predictive analytics (Campbell, 
DeBlois, & Oblinger, 2007; Macfadyen & Dawson, 2010) to include more fine-grained uses, in particular 
in the move towards lecturer- and student-facing analytics aligned with design for learning (Lockyer, 
Fleathcote, & Dawson, 2013). This movement has seen much progress within LA to achieve its goals of 
evidence-based improvements to learning, teaching, and the environments in which they occur 
(Siemens & Long, 2011). Part of this progress has come through novel sources of data and novel ways of 
acting based on analytics. A further part of this progress, and the focus here, comes from the 
development of new tools for obtaining evidence. We consider the development of new tools for LA to 
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be an example of design activity. One approach to designing tools is through abduction following the 
establishment of first principles (Gero, 2000), what we refer to here as theory-led design where these 
first principles are grounded in established theory. The claim that we make in this paper is that theory- 
led design has the potential to yield innovation in the development of LA tools and, in turn, that the 
development of LA tools and their use may contribute to learning theory. 

In this paper we present an example of theory-led design within one particular area of LA, the 
automated discourse analysis (ADA) of online collaborative learning. Collaborative learning can now be 
observed in synchronous or asynchronous learning environments (Garrison & Kanuka, 2004), as part of a 
physical, online, or blended modality (Bonk & Graham, 2012) and can range in group size from the small 
(i.e., two participants) to the very large (e.g., Kop, 2011). A commonality between these different forms 
of collaborative learning is that the discourse between participants can be analyzed to reveal 
information about the learning that is occurring (De Liddo, Buckingham Shum, Quinto, Bachler, & 
Cannavacciuolo, 2011; Mercer, 2007). As a result, discourse analysis has become "an important 
theoretical perspective for those concerned with the study of learning in social settings [in order to] 
examine ways which knowledge is socially constructed in classrooms and other educational settings" 
(Gee & Green, 1998, p. 119). In recent decades, many instruments have become available for 
conducting ADA to allow the natural language of group interactions to be parsed computationally and 
reveal aspects of the learning occurring, e.g., in the productive multivocality project, which identifies 
many different approaches to the analysis of group interactions (Suthers, Lund, Rose, Teplovs, & Law, 
2013). Any tool for analyzing group interactions necessarily makes ontological and epistemological 
assumptions about which phenomena are worth studying and the ways in which we may know about 
them (Lund & Suthers, 2013, p. 22). In the development of many tools for LA such assumptions may 
remain tacit. Theory-led design is one way of developing tools that allows for such assumptions to be 
made explicit and upfront. 

The paper is structured by first synthesizing a framework for theory-led design of LA tools based upon 
the literature. This framework is then used in a demonstration of the development of a novel tool to 
analyze a real life dataset from a group learning situation that has previously been analyzed manually. 
This real life dataset is in a subdomain of ADA that has received recent attention, for instruments that 
can aid instructors in the real-time orchestration of groups during learning through topic detection (e.g., 
Oshima, Matsuzawa, Oshima, & Niihara, 2013; Trausan-Matu, Dascalu, & Rebedea, 2014). Finally, in the 
discussion and conclusion we show that by developing the analytic technique from first principles, 
rather than applying an off-the-shelf tool, there is the potential to add to the theory of small group 
cognition and orchestration through the design of new instruments. 
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2 BACKGROUND 

2.1 Frameworks for Learning Analytics 

A number of frameworks have been proposed to guide the design and use of instruments and 
representations in LA. What follows is a brief review to portray current thinking on how the 
development of instruments and representations is conceived. The terminology of instrument is used 
here to refer to individual methods of analysis (e.g., sentiment analysis, topic detection) and the term 
representation is used to refer to the way that the results of analysis/analyses are presented (following 
Greller & Drachsler, 2012). We will use the term tool to refer to the combination of both instrument and 
representation. Any new developments of instruments and representations are open to ethical debate 
(Lund & Suthers, 2013), by asking questions of "who is defining the measures, to what ends, what is 
being measured, and who gets to see what data?" (Buckingham Shum & Ferguson, 2012, p. 5). Such 
questions can have more profound answers by first considering the question: Where do novel tools for 
LA come from? 

The design of instruments for analytics is often construed as having an intention to answer the 
questions of users so that decision making and action can take place (e.g., van Barneveld, Arnold, & 
Campbell, 2012; Verbert, Duval, Klerkx, Govaerts, & Santos, 2013). This is an implicit design philosophy 
of user-centred design (Abras, Maloney-Krichmar, & Preece, 2004) in which the needs of the user are the 
dominant consideration during development. The users provide the impetus for developing novel 
instruments; and if the instrument is useful for allowing the users to take positive actions then it has 
been successful. Whilst this is a valid and proven approach to developing novel tools, there are 
considerations outside of user needs that can enhance tool development. Two variations are proposed 
within the Information, Representation, Affordance, Change (IRAC) framework for LA (Jones, Beer, & 
Clark, 2013). The first is to suggest that any instrument for analytics ought to be continually developed 
and changed in response to user needs following implementation. This is "change," the C in IRAC, and it 
refers to the need for analytics to evolve over time based upon user and institutional needs. The second 
is to call for "mindful innovation" (Jones et al., 2013, p. 447) in developing tools, explained as a 
"nuanced appreciation of context" (p. 447), a call found elsewhere in the literature (Macfadyen & 
Dawson, 2012). However, much ambiguity remains as to how to achieve mindful innovation; and how 
user-centred design, continual change, and ethical concerns fit together. 

Another framework for LA is useful for establishing terminology. A general framework for LA was 
proposed by Greller and Drachsler (2012) that critically considers LA from the perspectives of 
stakeholders, limitations, objectives, instruments, and data. In this framework stakeholders are those 
who will make use of the representation resulting from the analytics, and are identified as learners, 
educators, institutions, and researchers. Objectives are identified through the broad aims of reflection 
and prediction. Many types of data can be utilized and instruments are used in conducting the analysis 
of these data with an acknowledgement of both intrinsic and extrinsic limitations. The framework 
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identifies the need to have clear objectives, especially with respect to stakeholders, in determining 
which instruments will be used and appropriate representations. 

2.2 Automated Discourse Analysis and Topic Detection 

This section provides a brief overview of Automated Discourse Analysis (ADA) and topic detection, areas 
where significant innovation in tools has occurred over recent decades. This background will be drawn 
upon in the example of developing a novel instrument in Section 4. Discourse analysis has been shown 
to be of significant utility to researchers, to learners, and to educators (Gee & Green, 1998). Over the 
past two decades a number of methods for automating discourse analysis through computation have 
been introduced to the study of learning, often through techniques originating in the field of 
computational linguistics (Chiu, 2008; Rose et al., 2008). Methods of ADA, when used to improve 
teaching and learning, form a part of the field of learning analytics (LA) when these methods are used 
for "the measurement, collection, analysis and reporting of data about learners and their contexts, for 
purposes of understanding and optimizing learning and the environments in which it occurs" (Siemens & 
Baker, 2012, p. 1). ADA has the potential to aid researchers to study learning, learners, and learning 
environments (Thompson, Ashe, Carvalho, et al., 2013); aid students by developing student-facing LA for 
meta-cognitive feedback and formative assessment (De Liddo et al., 2011; Lockyer & Dawson, 2011; 
Buckingham Shum & Ferguson, 2012; Wise, Zhao, & Hausknecht, 2014); and aid educators with 
educator-facing analytics, some of which can help facilitate orchestration (De Liddo et al., 2011; 
Trausan-Matu et al., 2014). Researchers utilizing ADA techniques for LA are concerned not just with 
mining data for analysis, but with using this analysis to enhance learning or learning environments in the 
subsequent redesign of learning tasks or for learner-facing analytics (e.g., Scheffel et al., 2012). 

Thus far, much of the work in ADA has focused on the analysis of learning events after they have 
occurred — a focus on making claims about the learning (Lund & Suthers, 2013) rather than upon utility 
for in-class orchestration (Dillenbourg & Jermann, 2010). There is significant potential for further 
developing the use of ADA in the orchestration of classroom environments whether face-to-face or 
online in real time (Trausan-Matu, 2013; Trausan-Matu et al., 2014). The introduction of technology into 
face-to-face environments and the removal or distancing of the instructor in online collaborative 
settings has resulted in a complex web of challenges to the traditional management of these groups. As 
such, a tool that could give instructors a quick overview of discussions in progress, and track transitions 
through expected phases of group collaboration would be valuable. An example of such a tool is seen in 
KBDeX (Matsuzawa, Oshima, Oshima, Niihara, & Sakai, 2011; Oshima, Oshima, & Matsuzawa, 2012), 
which builds a network from group interactions and provides a visualization of group activity based 
upon social network analysis. 

Automated topic detection is concerned with the discovery of key topics, themes, or concepts from 
within a body of text (Blei, Carin, & Dunson, 2010). Topic detection takes a corpus of text as an input, 
performs automated analysis, and returns a set of themes as an output. An early example is the "word 
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cloud" approach, a simple listing of the words most repeated within a discourse (Cui et al., 2010). Far 
more advanced tools have been developed since, such as Latent Dirichlet Allocation (LDA), keyphrase 
extraction algorithms (Witten, Paynter, Frank, Gutwin, & Nevill-Manning, 1999), and hierarchical LDA 
models (Wallach, 2006). Recent research in computer supported collaborative learning (CSCL) has 
utilized these approaches for the analysis of collaborative problem solving (Trausan-Matu & Rebedea, 
2010), classification of online discussion (Mu, Stegmann, Mayfield, Rose, & Fischer, 2012; Trausan-Matu, 
Dascalu, & Rebedea, 2012), visualization of chat (Trausan-Matu, Rebedea, Dragan, & Alexandru, 2007), 
marking participation in online chat (Rebedea, Dascalu, Trausan-Matu, Armitt, & Chiru, 2011), evidence 
of metacognition (Chiu, 2013), and the extraction of scientific concepts from student interviews (Sherin, 
2013). These examples use a probabilistic approach (most typically LDA) to build a model from the text 
that can then be used for analysis (Blei, Ng, & Jordan, 2003). 

One limitation of this approach is the need for a large corpus of text from which to build an initial 
statistical model. A more recent approach to topic detection that does not have this limitation is the 
graph analytical approach. Topic detection in this way proceeds by constructing a graph from utterances 
in discourse, which can then be mined to give information about the topics (Sayyadi & Raschid, 2013). 
The advantage of the graph analytic approach is that it can identify relevant text from small samples of 
the discourse. A second advantage of this approach is that the salient features of the resulting graph can 
then be represented in the output (e.g., Oshima et al., 2012). Recent work has begun to combine the 
two approaches (Teplovs & Fujita, 2013). 

2.3 CSCL in the Context of ADA 

This section provides some background on the theory of CSCL in the context of ADA, which we will draw 
upon in the example of developing a novel instrument in Section 4. The example in Section 4 involves 
design for instructors orchestrating online group cognition tasks. Analysis of collaborative learning has 
been built upon several seminal theories including, amongst others, the sociocultural theory of Vygotsky 
(1930/1978), communities of practice (Lave & Wenger, 1991), conversation analysis (Sacks, 1972), and 
the role of computers in cognition (Winograd & Flores, 1986). Many of these theoretical perspectives 
(Stahl, Koschmann, & Suthers, 2015) identify cognition as being "distributed across people and tools, 
situated in contexts, within small groups, involved in larger activities and across communities of 
practice" (Stahl, 2012, p. 1). From these foundations, theory has been developed to support the 
research of individuals working in groups, small group cognition, and community cognition (Stahl, 2012). 
One area of recent focus has been group cognition (Stahl, 2006), which assumes that knowledge 
emerges from the interaction of social organization, tool use, and epistemic factors (Goodyear & 
Carvalho, 2014), which are beyond that of the influence of any particular individual. In addition to the 
theories of CSCL, two other bodies of research are important to this work: 1) the role of time and order 
in analyzing the activity of learners (Goodyear, Jones, & Thompson, 2014; Reimann, 2009); and 2) 
classroom orchestration (Dillenbourg & Jermann, 2010; Mazzolini & Maddison, 2003). The term 
classroom orchestration is widely taken to refer to "the design and real-time management of multiple 
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classroom activities, various learning processes and numerous teaching actions" (Dillenbourg & 
Jermann, 2010) and it brings together many of the theoretical perspectives mentioned above. 
Educators, tasked with the responsibility of orchestrating learning across multiple online or in-person 
groups, in open, flexible and digitally enabled learning environments, face numerous challenges at 
multiple scales that require a form of omnipresence to provide timely and effective intervention — 
intervention that is often critical to the successful completion of group work (Mazzolini & Maddison, 
2003). Whilst there is debate about the nature, role, and scale at which orchestration should be 
deployed, there is general agreement about the need for technologies that support those who teach in 
technology rich environments (Roschelle, Dimitriadis, & Hoppe, 2013), with a focus on functionality that 
enables those who use it to monitor activity and adapt elements of the design accordingly, in real time 
(Dillenbourg, Nussbaum, Dimitriadis, & Roschelle, 2012; Martinez-Maldonado, Clayphan, Yacef, & Kay, 
2015). 

In this paper, we concentrate on the perspective presented by group cognition, using a conversation 
analysis approach in order to use learning analytics in tools to support orchestration. The speech 
produced within a group is influenced by factors such as the other members of the group, the nature of 
the task, and the tools, both digital and physical, in the learning environment. 

3 THEORY-LED DESIGN OF ADA TOOLS 

3.1 Designing from First-Principles 

What then, given this background, is the role of theory in the "mindful design" of new tools? In many 
cases, new tools come to be developed from existing tools. Consider that an LA practitioner (henceforth 
"researcher") has clear objectives, data, stakeholders, and limitations. In most situations, many available 
instruments could be used to address these objectives. Whilst researchers will often adopt an existing, 
validated tool from the existing corpus, there may also be potential to design a new tool based upon 
existing methods and first principles. If researchers are designing tools, then the processes by which 
novel methods are developed are of considerable interest. Such activity to produce novel methods is 
termed innovative design in that it requires researchers to move beyond the grammatical application of 
existing designs (Gero, 2000). 

Innovative design can be described through one or more of five processes, and the category of theory- 
led tool design fits into the first of these. Processes that rely on first principles involve the use of 
abduction from a set of "causal, qualitative, or computational knowledge" (Gero, 2000, p. 194). 
Processes of analogy involve the transfer of elements of structure or behaviours from one problem or 
domain into another problem or domain. Combination involves the "addition of two sets of ideas or 
some subset of them" (p. 191). Transformation is the "alteration of one or more structure variables by 
an external process" (p. 191). This involves altering a method to make it suit the objectives. Emergence 
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is the process whereby "extensional properties of a structure are recognized beyond its intentional 
ones" (p. 193). 

Some of these processes for innovative design can be demonstrated by discussing the recent example of 
the PolyCAFe system (Trausan-Matu et al., 2014). The PolyCAFe model automates the analysis of 
unmoderated chat conversations, aiming to assist educators in orchestrating and assessing groups 
engaged in potentially hours of chat dialogue. In this model, the researchers cite Bakhtin's discussion of 
the dialogic imagination and evocation of polyphony (e.g., counterpoint in music) (Bakhtin, 1981/2010) 
as the basis for their choice of how to analyze and represent the arguments that participants are making 
within a dialogue. This is an example of the use of first principles, beginning with the theory and using 
this to suggest function, structure, and behaviours of the tool. Trausan-Matu et al. also use analogy in 
drawing upon the experiences of tutors described in Stahl's study of virtual maths teams (Stahl, 2009) 
and set the task of making an automated analog of what the tutors were doing. They use combination in 
combining the existing techniques of Social Network Analysis (SNA) and Latent Semantic Analysis (LSA) 
in developing a composite metric and representation. 

3.2 Mindful Design from Theory 

Theory-led design is defined as a form of innovative design that begins with the use of theory to 
establish first principles, and proceeds towards the development of a tool through an abductive process. 
To be useful to practitioners, however, a more pragmatic description than this is required: For example: 
What are these first principles? How are they developed and how are they used? These questions are 
addressed by proposing a general framework for the design of new tools. The framework refers to the 
distinctions of function, behaviour, and structure (Gero, 1990), and first principles fit into this framework 
in establishing function during the design of the tool. 

The function of an artefact is its constructed teleology — the intentional purpose for undertaking the 
design activity (Dorst & Vermaas, 2005; Gero & Kannengiesser, 2004). For example, the function for 
designing an office chair could be described as "supporting a human at rest whilst they work at a 
computer workstation." This function then forms the basis for establishing the expected behaviours of 
the artefact. These are the ways in which the artefact's satisfaction of the function can be measured. For 
example, an office chair might have behaviours that relate to safety, ergonomics, cost, and aesthetics. A 
structure is then developed as the elements composing the artefact and their relationship, e.g., size, 
shape, material, and relationships of parts. This designed structure can then be evaluated to determine 
the extent to which the expected behaviours are satisfied — these are the actual behaviours. Each of 
these steps — establishing function, expected behaviours and structure — may be revisited in the 
process of reformulation, showing design to be an iterative process. 

The utility of this description of the design process is to allow for distinction between cognitive tasks 
that take place during design (Gero & Kannengiesser, 2004). For example, there is nothing in the 
description of function that directly suggests structure; it is only through behaviours that function 
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influences the structure. This description of design is not intended to be prescriptive nor to suggest that 
design occurs in a linear way, but rather be a useful ontology for discerning, analyzing, and studying the 
different processes involved in design (e.g., Williams, Yoon Suk, Gero, & Paretti, 2012). Figure 1 captures 
this process of design as it relates to the theory-led design of tools for ADA of group learning. 

3.2.1 Establishing Function 

Design of LA tools from first principles requires that time be spent establishing the function for the tool 
being developed. In this step an engagement with theory influences the design process and is the 
foundation for innovation. There are five domains in which theoretical decisions are made during 
analysis of group discussion (Wise & Paulus, in press): 1) ontological and epistemological foundations; 2) 
the way that learning is conceived; 3) the kinds of claims expected to be made; 4) data sources; and 5) 
data analysis methods. Each of these presents an opportunity to influence the function in the direction 
of innovative design. A designer can undertake an interrogation of their assumptions by engaging with 
the literature that relates to their understanding of the world, the nature of knowledge, the nature of 
learning, the outcomes that will come from the use of their tool, the sources from which their data will 
be drawn, and ways to analyze that data. 

In this paper we are concerned with the development of tools for ADA. In ADA the theory that informs 
first principles pertains to the way that groups learn together, the language used by groups during 
collaboration, and what that language signifies. The function relates to the objectives for the tool (what 
can be done with this tool that could not be done previously?) and to the stakeholders (who will be 
using this tool?). An example of this process, and specific relevant theory, is detailed in Section 4. 

3.2.2. Establishing Behaviour and Structure 

The expected behaviours of the tool are developed from an understanding of function grounded in 
theory. An example in the domain of LA serves to illustrate the use of function to develop behaviours 
and structure. If the function of the Course Signals learning analytics tool was perhaps to "use available 
data to support students in their learning" then the expected behaviours of the tool are those things 
that the designers consider relevant to measure in order to establish the achievement of this function 
(Arnold & Pistilli, 2012). One indicative measure might be an increase in student retention (Arnold & 
Pistilli, 2012). An example of another expected behaviour could be indicators of engagement by 
lecturers (e.g., number of views) with the Course Signals tool. The expected behaviours established do 
not specify structure; rather they specify the way in which structure will be measured. 

In the context of ADA of group learning, behaviours relate to measures of the tool's use and operation 
and of its impacts upon those who will be using it (e.g., researchers, academics and students). The use of 
theory in developing the function for the tool allows for the designer to challenge pre-existing and often 
implicit beliefs about behaviours for the tool. 

The structure of the tool is developed in an attempt to create an artefact that will satisfy the expected 
behaviours. In a tool for ADA of group learning this involves decisions about data sources, methods of 
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analysis, and methods of representation (Wise & Paulus, in press). The use of theory in developing the 
function may again serve the designer in opening them up to the potential for developing novel 
elements of structure. There is much potential for innovation within ADA that comes from novel sources 
of data and from the development of new forms of analysis and new modes of representation. 


3.2.3 Iterating through Evaluation and Reformulation 

In this framework for design, iteration occurs through one of three types of reformulation: 1) where 
structure is revised; 2) where expected behaviours are revised; and 3) where function is revised (Gero, 
1990). A trigger for reformulation is evaluation, where expected behaviours of the structure are 
compared to actual behaviours. In some cases the designer can change the structure to produce the 
desired behaviours. For example, in reformulating structure, the designers of the Course Signals 
software might change their algorithm or their data sources to improve performance of the tool. In 
other cases, the behaviours of the design might be changed through reformulation. For example, in 
reformulating behaviours, the designers of the Course Signals software might decide that the behaviour 
of student retention is not appropriate for achieving the desired function and other behaviours could be 
introduced. Finally, the function of the tool can be revised — perhaps with what has been learnt through 
the process of designing, the original intent is no longer a good description of the purpose of the design 
activity and is reformulated. 



Figure 1. A general framework for design of ADA instruments and representations from first 


principles. 


4 DEVELOPING AN INSTRUMENT FOR TOPIC DETECTION IN 
ORCHESTRATION 


This section describes the development of a tool for topic detection through ADA as an example of 
theory-led design. Referring to the framework developed in Section 3, it describes the processes of 
establishing function, establishing structure and behaviour, and evaluation and expected reformulation. 
The design produced can be summarized as a tool for instructor-facing topic detection using graphical 
modelling, where heuristics for parsing natural language are used to construct a graph that is then 
mined for data. 
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4.1 Establishing Function 

The function of the tool developed was to aid instructors in the orchestration of online collaborative 
learning tasks. The process for arriving at this function is described here, including the theory informing 
first principles. Two overarching theories supported this work. Group cognition recognizes that learning 
can be observed at the level of small groups and communities (Stahl, 2006). Adoption of the theory of 
group cognition shifts the conception of learning away from individual minds towards the way that small 
groups or communities work together to create shared understanding. The focus is upon the 
development of the constructs adopted by the group as they develop a "joint problem space" revealed 
through the sequence and semantics of their discourse (Stahl, 2006). Within this understanding of group 
cognition, Reimann's theoretical work informs the understanding of data analysis in terms of the 
importance of time (Reimann, 2009; Reimann, Markauskaite, & Bannert, 2014). Many types of 
automated analysis of discourse (especially techniques based on probabilistic models) ignore vital 
information about the timing and process. A contribution of this theory was to question whether further 
use might be made during analysis of the sequence in which the words and utterances of group 
discussion occur. 

The initial impetus for developing the tool occurred whilst carrying out a number of studies using 
manual discourse analysis. The objective of the manual studies was to understand the complex interplay 
between aspects of the learners' social and epistemic activity and the activity associated with the tools 
and physical and digital learning environments. Two streams of data were related to the discourse. The 
first was the Design Ideas code (Thompson, Ashe, Yeoman, & Parisio, 2013), which was the identification 
of ideas important to the development of each group's design; the second was the CPACS code 
(Kennedy-Clark & Thompson, 2013a). CPACS (the Collaborative Process Analysis Coding Scheme) is a 
multi-level coding scheme based on coding systems used by Mazur and Lio (2004), Sawyer (2006), Nivre, 
Allwood, & Ahlsen (1999), and Soter et al. (2008), focusing on features of speech and macro (action and 
content) and micro (attitudinal, tense, modality, and pronouns) elements of turn taking (Kennedy-Clark 
& Thompson, 2013b). The CPACS code includes social (phatics, salutations, leave taking), planning, topic, 
task, tool use, and off task (Kennedy-Clark & Thompson, 2013a). Other related work examined the 
presence of oscillation between consecutive corresponding points of the same phase of a problem in the 
contents of discourse as an indicator of successful collaboration (Kennedy-Clark & Thompson, 2013b; 
Thompson, Ashe, Yeoman, Parisio, 2013). These studies provided a basis for further investigation into 1) 
the potential for using language in group discourse to aid in detecting successful collaboration; 2) the 
role of turn taking, in particular, during collaboration. 

It was observed that the key concepts in the Design Ideas study that participants referred to were 
primarily nouns, and that there was a repetition of words (echoing) between participants around key 
topics. These nouns were mapped to the content codes: social, tools, planning, topic, and task. In the 
original, manual studies, the identification of these nouns and subsequent mapping allowed the authors 
to identify patterns of collaboration that mapped to phases in the group's design work (orientation, idea 
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generation, social interaction, and technology failure) (Thompson, Ashe, Yeoman, Parisio, 2013). 
Continual referral to the keywords concerned with the topic and the task indicated that students were 
integrating specific content knowledge in the generation of ideas for the design task. The identification 
of any of these phases, be they of progress (orientation and idea generation) or a lack of progress 
(negative social interactions and technology failure) is of interest to an instructor. 

Further, the observation was made that repetition of words is a key indicator of importance. This was 
supported by further reading of the literature, with Tannen (1987) suggesting that repetition serves 
many purposes in speech, both within an utterance and in the interplay between participants. For 
interplay, "repeating the words, phrases, or sentences of other speakers (a) accomplishes a 
conversation, (b) shows one's response to another's utterance, (c) shows acceptance of others' 
utterances, their participation, and them, and (d) gives evidence of one's own participation" (Tannen, 
1987, p. 61). These are valuable indicators of which words might correspond to important topics and 
provides an indicator of a technique that may supplement or extend existing measures that are 
commonly used, such as LDA. A feature of human language in conversation is that segments of the 
previous utterance are often repeated, to confirm common ground and establish or alter meanings 
(Norrick, 1987; Tannen, 1987). This is highlighted by Norrick who recognizes that "speakers not only 
repeat their own words and phrases at the level of the turn, and their own turns at the discourse level, 
but they also echo the wording, rhythms, and turns of their interlocutors" (Norrick, 1987, p. 245). 

These underpinnings informed the function for the design task. Assumptions can be made explicit: that 
the tool is concerned with groups collaborating online; who use language whilst learning collaboratively; 
in the presence of an instructor. The concepts that the group members make reference to are likely to 
be of interest in determining whether the group is discussing the task and the topic. Further, the turn¬ 
taking of participants likely reveals something about which parts of the discourse are important. 

Given this basis, the instructor was identified as the key stakeholder. The objective was to create a tool 
that could assist the instructor in knowing which groups are on or off task. A reading of existing tools 
showed that there is much potential for innovation in this area in the data displayed to instructors and 
the way that it is represented. 

4.2 Establishing Behaviour and Structure 

Having established this function of enabling instructors to quickly understand the status of multiple 
group collaborations in real-time, the expected behaviours of the tool could be developed as the 
measures that would indicate that this function had been achieved by structure. An important 
behaviour was that the tool had to be perceived as useful by instructors in revealing something about 
the status of groups. This behaviour cannot be measured directly from the structure (from the tool 
itself) but rather requires measurement of how it is used by the stakeholders. 
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A behaviour that did not require measurements of stakeholders was that the tool would show the 
current group constructs relating to the task and to the process of designing. This behaviour can be 
considered a measure of "how well does the tool reveal the current status and phase of design of the 
group?" If this behaviour were satisfied, the tool could assist orchestration by suggesting which phases 
of design each group was in "at a glance." During the process of design, this behaviour was decomposed 
into smaller behaviours as a method for establishing the behaviour. A further behaviour was that the 
tool could be used by non-experts and this required that the visualizations created by the tool would 
enable a non-expert to make decisions on-the-fly during orchestration of online groups — that the 
visualizations should show change over time and that the tool would reveal something about group 
dynamics through the language, in particular through word repetition. 

Data visualization was deemed important due to the complex nature of the task (a design task about a 
specific topic), the tools and unique setting, and the requirement to work collaboratively. There is much 
in the literature about the ability of non-experts to interpret representations, which lies outside the 
scope of this paper. In this case, we followed Trausan-Matu et al. (2014) in utilizing visual 
representations of words in the discourse to aid non-expert comprehension, and Oshima et al. (2013) in 
using a graph structure to make explicit the relationships between words. For classroom orchestration 
occurring in real-time, the discourse from different groups makes up a continuous data stream. An 
educator would need to focus on comprehending the most recent data in the context of the entire 
discourse (rather than simply on the entire discourse). Making a decision about structure, a sliding 
window was deemed to be an appropriate representation, in which results from the most recent x 
number of utterances or time period is displayed, and changes to this over time can be observed. From 
the manual studies it was estimated that the window in this type of task should span approximately 50- 
100 utterances to show where negative social interactions and off-topic work was taking place. 

Given these behaviours, the structure was developed through a process of synthesis: the cycle of 
developing structure, evaluating actual behaviours and reformulating the structure. Some behaviours 
were challenging to measure (e.g., how useful the tool was to instructors) and were not included in the 
cycle of synthesis. Other behaviours were used: the perceived ease of at-a-glance understanding of the 
visualizations produced; the perceived value of the concepts picked up by the tool for revealing the 
status of the group and the variables that went into this. Pragmatically, this latter required the most 
innovation, with many variables to consider, such as size of the window of utterances, the analysis of 
language to utilize, the relevance of sequence of utterances. A detailed description of the structure is 
useful both for allowing others to replicate the tool but also for understanding how these behaviours 
were developed into a working, innovative prototype for a tool. 

4.3 Describing the Structure of the Tool 

The structure of the technique was established through this cycle of synthesis. The tool is described with 
sufficient detail that it could be replicated. The tool adopts a graph-based approach to representing 
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group dynamics, in which the relationships between words (in particular repetition between members) 
are used in drawing the graph. The resulting visualizations are then used by instructors for orchestrating 
group activity. Whilst the idea of using a graphical, heuristics-based approach to topic detection is not 
new, the desire to include repetition in the analysis meant that appropriate technology was not 
available, and the technique to do so would have to be developed. The technique will be described in 
three parts: 1) the NLP pipeline in preparing the natural language and extracting the nouns for 
processing; 2) drawing the graph based upon relationships between nouns; and 3) analyzing the graph 
and developing a representation based upon the relationships between words. The example provided 
here is being conducted using real-world data from a study of learners engaging with design tasks 
(described in Thompson, Ashe, Yeoman, & Parisio, 2013). 

4.3.1 The NLP Pipeline 

Many NLP approaches have in common a pipeline of automated preparation of the natural language 
data prior to analysis (Manning & Schiitze, 1999). In this work, an NLP pipeline was established to clean 
the data and move from a natural language utterance to a list of nouns from those utterances for 
further processing, shown in Figure 2 of the pipeline. These techniques have been used previously in 
analysis of CSCL (Mu et al., 2012; Rebedea et al., 2011; Trausan-Matu, Dascalu, & Rebedea, 2012). The 
process was implemented in the Python language using a combination of the Natural Language Toolkit 
(Bird, Klein, & Loper, 2009) and the pattern.en natural language module: 1) data preparation, 2) 
tokenization, 3) parts of speech tagging, 4) stopword removal, and 5) word stemming. The outcome 
from the NLP pipeline is a list of nouns from the text. An example of outcomes from the NLP pipeline is 
shown in Table 1. As the pipeline has an input of natural language, there are likely to be frequent 
instances of typographical errors, which can lead to an output with the same errors. 



tokenized 


processed data 

utterance 


(special characters, punc¬ 
tuation, preparation) 


PoS tagging 


nouns 


stopword removal 
stemming 


word-list 


word graph construction 
network analysis 
(eigenvector centrality) 


Figure 2. Natural language processing pipeline of utterances 


Table 1. Utterances and outcomes from NLP pipeline graph construction 


Line 

Author 

Utterance 

Outcome from NLP pipeline 

68 

Alfred 

I've got a plain book, we've got the science 
syllabus, the geography syllabus... 

book, science, geography, 
syllabus 

69 

Bronwyn 

Put the music on, see it works fine. 

music 


4.3.2 Graph Creation 

The technique functions by creating a graph within which words are nodes and relationships between 
them are edges. In drawing the graph the relevant questions are 1) which connections are meaningful? 
and 2) how much weight should be given to these connections? The resulting graph can subsequently be 
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mined for information using a suite of algorithms from the field of network theory (Wasserman & 
Galaskiewicz, 1994). 

In this work, the graph (Figure 3) is constructed using the words extracted from the chat dialogue. Edges 
are drawn between these words based on 1) co-occurrence and 2) adjacent occurrence by different 
actors. Co-occurrence refers to the identified words that occur within the same utterance. Their 
relationship is indicated by having been uttered within the same sentence. Adjacent occurrence refers 
to words in proximal utterances. A further criterion is that only adjacent occurrences by different actors 
are considered. 

The resulting graph produced by this process is a representation of the relationships between the 
pruned words. Edges in the graph are weighted to capture information about the frequency of 
connections. For example, if the same extracted word occurs within three separate utterance times then 
it means that this has occurred more than once. Equation 1 shows the calculation of weighting of the 
edge, as term frequency (the number of times the same extracted word has been used in the utterance), 
over the number of terms in the utterance (the total number of extracted words in the pruned list). Two 
distinct benefits are derived from this weighting. First, it prevents the length of each utterance from 
having excessive influence on the final graph; second, it allows for the additive development of weights 
on connections as new utterances are processed. 

These rules can be formalized as a group chat T made up of a number of utterances U, Equation 1, 
where each utterance consists of a number of extracted words k: 



( 1 ) 


( 2 ) 


Each extracted word, k, becomes a node within the graph. An edge ey is drawn between every pair of 
extracted words k xi , k x j that co-occur within the utterance U x made by actor a: 


^ x,i F' k x j Si j 


(3) 


The weighting given to each edge is inverse to the number of terms within the utterance U x which will 
be referred to as p: 


An edge is also drawn between extracted words where an adjacent occurrence occurs and there is a 
different actor b such that a ^ b: 



(5) 
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In the case of these adjacent utterances this takes the number of terms in the subsequent utterance 
multiplied by the previous weight: 

...adjacent _ 1 w _vrevious :r\ 

w e i:j - ^current X P l b ) 

Weights are summed where more than one connection is being made, as occurs where a word has more 
than one edge joining it, such as in cases where the same word appears in multiple utterances 
throughout the conversation. 

The two phrases in Table 1 can be used to provide an example of the construction of the graph. The two 
utterances are adjacent and are uttered by different actors. The resulting nodes, edges, and weightings 
are represented in Figure 3. The utterance in U 68 results in the creation of weighted links between the 
words, "book," "science," "geography," and "syllabus." As there are four extracted words in the 
utterance, each link has a weight of 0.25. In the second utterance in U 69 there is one word identified, 
"music." As the second utterance was adjacent and uttered by a different actor, links are created 
between the two sets of words with weightings of 0.25. 



Figure 3. Construction of weighted graph of extracted words and relationships from the 

utterances in Table 1 

The composition of the graph resulting from this construction of CSCL data is then analyzed to 
determine topics, using the centrality of each word within the graph as an indication of its importance 
(Wasserman & Galaskiewicz, 1994). Due to the way that the graph has been constructed, the 
importance of the word as indicated by centrality incorporates factors of frequency of the word, its 
relationship to other words, the importance of those words, and the relationship between actors. 

The centrality measure used in this work is eigenvector centrality which is typically used to measure 
influence within the network (Bonacich, 1972). This algorithm is more widely known as the technique 
upon which the Google PageRank is based (Page, Brin, Motwani, & Winograd, 1999). This measure finds 
the influence of each node upon the entire network, where the importance of a node is affected by the 
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importance of each other node that connects to it. In our model the eigenvector centrality measure is 
used to indicate the words most influential in the graph. 

4.3.3 Representation 

The representation of the data provides the reader with information about how the key concepts relate 
to one another. In this demonstration, the most recent 50 utterances were used. The key concepts used 
by each group are shown as nodes, with the width of the edges representing the strength of 
connections, Figures 4-9. This provides an intuitive understanding for the viewer (in practical use, the 
instructor), where a richness of words indicates significant activity, whilst a paucity of words indicates 
parsimonious contributions. 

Figures 4-13 below are the representations generated using the graphical analysis technique for a 
sample of the utterances generated by Group 1 and Group 2 (from utterance 300 to utterance 449). 
During the period represented in what follows, the group is undergoing a phase of idea generation 
prompted by the presence of the facilitator. The analysis of the processes of idea generation and 
collaborative discussion in Group 2 has also been reported on in previous papers (Thompson, Ashe, 
Wardak, Yeoman, & Parisio, 2013; Thompson, Ashe, Yeoman, Parisio, 2013). During the collaborative 
design task, Group 2 underwent a period where the processes of collaboration broke down due to a 
failure in the technology they were using, and which was preceded by negative social interactions. It is 
this point, as the group is fracturing, that is represented by the selection of utterances below. There are 
five graphs for each group. While each graph represents 50 utterances, there is an overlap of 25 
between each graph. This is done to show the gradual change in the discourse over time (i.e., the idea 
being that in practice this would be a moving window of 50 utterances that the educator can view at any 
time). First we will use a rich description, based on the video data, of what was happening in each group 
at this time. Then we will show how this relates to the graphs. Finally, we will provide a comparison 
between the two groups. 

Group 1 spent much of their time negotiating ownership of tools and space. The facilitator attempts to 
move them from the phase of negotiation and orientation to idea generation before they have resolved 
the roles of the group members and the rules of the group. Each time the facilitator leaves the group, 
they revert to this negotiation and orientation phase. In Figure 4, the key concepts with strong 
connections between them are Facebook, game, and guy. The group is in the middle of a period of 
ideation in which the facilitator is leading them through the development of ideas for the game. He is 
asking them to empathize with the future users of the learning resource, with reference to guy and 
person and asking them to find connections to other ideas. 

In Figure 5, the discussion about ideas for the game continues, mainly led by the facilitator. However, as 
can be seen in utterances 350-399 (Figure 6), the network of key concepts is highly decreased. It is at 
this point that the discourse slows as the students are recording the idea in their shared document, and 
having minor technical difficulties with the mouse. Conversation is stilted as the facilitator tries to evoke 
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responses from the students as to how they envision their learning resource, as their concentration is on 
the technology. 

Figure 7 shows the beginning of the group discussing their own ideas that have come from the 
conversation with the facilitator. There are many different ideas being thrown around, as evidenced by 
the many key concepts without strong connections between them. Figure 8 shows the connections 
becoming stronger as the group comes to a shared understanding of the proposed idea. 



Figure 4. Representation of group 1 collaboration, utterances 300-349 
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point 


school 


tool 

constraint 


Figure 5. Representation of group 1 collaboration, utterances 325-374 



Figure 6. Representation of group 1 collaboration, 
utterances 350-399 


Figure 7. Representation of group 1 collaboration, 
utterances 375-424 
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At the same time as the idea generation phase could be observed in Group 1, Group 2's discussion was 
characterized by negotiation of the rules and roles within the group of five boys (e.g., smarty, pants) and 
reference to the creation of a Word document in which one member catalogued the group's earlier 
ideas (e.g., spelling, mistake, word), and the eventual descent into a long discussion that was classified 
as off-task (e.g., muffin, icing, cherry, cupcake) in reference to drawings being made on the writeable 
wall by a different group. 

Figure 9 represents the part of the discourse in Group 2 just before the collaboration breaks down. The 
students have just begun discussing an idea related to a detailed element of the game. However, they 
are drawing at the same time, and so the discourse is stilted. Figure 10 shows the beginning of the 
negative social interactions, with negative comments about spelling mistakes, and name calling ( smarty 
pants). Figure 11 shows one student's comments about a drawing. At this point there has been a failure 
in the technology, and most of the discussion is about action to fix the problem (with few nouns). 
Figures 12 and 13 show the breakdown of the focus of the group, as they are distracted from the task 
and have an off-task conversation (e.g., muffin, icing, cherry, cupcake) in reference to drawings they 
were making on the writeable wall. 
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Figure 9. Representation of group 2 collaboration, utterances 300-349 




Figure 10. Representation of group 2 Figure 11. Representation of group 2 

collaboration, utterances 325-374 collaboration, utterances 350-399 
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Figure 12. Representation of group 2 
collaboration, utterances 375-424 


Figure 13. Representation of group 2 
collaboration, utterances 400-449 


In both groups, phases in the process of idea generation could be identified by the patterns of key 
concepts and the nature of the connections between them. Few key concepts with strong connections 
indicated a clear idea; however, the nature of the representation did not show whether it was one 
person making their case (as in Group 1, Figure 4) or an occurrence of shared meaning making (as in 
Group 1, Figure 8). Many key concepts with limited strong connections indicated a period close to 
brainstorming, with many participants suggesting ideas, most unrelated to each other (as in Group 1, 
Figure 7). Few key concepts with few connections indicated action of some kind, whether that was the 
recording of ideas (as in Group 1, Figure 6), or troubleshooting (as in Group 2, Figure 11). Finally, key 
concepts unrelated to the task or topic indicated off-task discourse (as in Group 2, Figure 12) and an 
increase in links between key (off-task) concepts indicated an extended period of off-task work (as in 
Group 2, Figure 13). In each of these cases, understanding what preceded the moments that each graph 
represents helps to understand the process that each group is undertaking, and in fact, in most cases 
cause for intervention would be the continuation of a group in any of the above phases of their 
collaborative design work. 

4.4 Evaluation and Reformulation 

As described in the framework, the tool requires evaluation of how the actual behaviours of the 
structure perform against its expected behaviours. Flowever, only some of the actual behaviours have 
been tested at this point due to limitations of time and resources. This section describes the process of 
evaluation and reformulation required to advance the design process further. 
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Most critically, evaluation is required for the behaviour of how useful the tool is to instructors. This 
would ideally be done as an in-situ experiment, with a number of instructors that currently orchestrate 
online group activities using the tool for LA and being surveyed about their perceptions of the utility of 
the tool: Did it reveal anything about the status of the groups? Were the concepts indicated by the tool 
indicators of key concepts used by the group? Was the tool reliable and/or useful? Could it be improved 
and if so, how? By asking these questions of multiple instructors, an understanding of how well the tool 
satisfies this behaviour could be developed. 

This understanding would most likely lead to reformulation of one or another of the three types. For 
example, the tool may prove to be entirely unhelpful to instructors, at which point the very function of 
the tool may be questioned and reformulated. Or perhaps the tool may prove useful but a behaviour is 
identified during the process of evaluation that could further improve the tool, leading to reformulation 
of behaviours, which in turn leads to changes in structure. Or some tweaks to the structure might be 
identified to improve performance in the behaviours already established. 

5 DISCUSSION AND CONCLUSIONS 

This paper has contributed an overview of different ways to develop novel tools for use in learning 
analytics and has focused upon design from first-principles as a way of promoting innovation in the 
design of tools for ADA. In Section 3 a framework for how theory fits into design of new tools was 
posited, and in Section 4 an example was presented that resulted in the creation of a new tool to aid 
instructors in orchestrating online groups, a tool that has potential significance (following future testing 
and validation) in its own right. The example shows how theory-led design occurs; yet even in the 
example it is apparent that the notion of design being theory-led lies in a continuum rather than a 
dichotomy. On this continuum the design of tools may be increasingly informed by theory. This is not 
opposed to design for meeting the needs of users but rather a complement. In the framework it was 
stressed that use of theory is most critical in establishing first principles, where theory allows for the 
epistemological foundations for the tool and the perspectives upon learning adopted to be made 
explicit. 

The example developed in Section 4 makes explicit a commitment to group cognition and language 
analysis, as well as an understanding of the role of word repetition in turn-taking during conversation. 
These principles informed the development of the novel tool — the selection of data, the method of 
analysis and of representation. The tool itself was demonstrated in Section 4.3 using real-world data 
from natural language chat between participants in a two groups engaged in a design task. Whilst the 
results show the proof of concept of the tool, much work is required to demonstrate its utility within the 
cannon of LA. In particular, the claim that it is useful for instructors in orchestrating online collaborative 
work needs be tested, ideally with real-world instructors of online classrooms in real-time, testing both 
for demonstration of the tool's accuracy in helping instructors and for instructors' perceptions of its 
usefulness. 
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The technique developed provides a visual representation (for the purposes of orchestration) of 
automatically identified topics raised in the discourse using sliding windows (to show change over time). 
Other research discussed (Thompson, Ashe, Wardak, et al., 2013) showed that the development of 
design ideas, which can be identified using this technique, was an effective indicator of a group's 
progress through this part of the task, and also aligned with other areas of activity, such as collaborative 
processes. With training, we propose that this technique could be used to provide instructors with 
visualizations that could indicate a group that needs help to progress in their task. We do not suggest 
that this would be the only measure, indeed other parts of speech can be used to indicate successful 
collaboration (Thompson, Kennedy-Clark, Kelly, & Wheeler, 2013). In the other research on this dataset, 
we suggested that it is the social activity of the learners that needed to be supported by the facilitator, 
that the learners were very capable of generating ideas, but that unproductive pauses, such as negative 
social interactions, or an inability to move past the negotiation of roles and rules in the group, 
prevented the groups from engaging in this idea generation. Understanding how the development of 
successful collaborative activity can be supported by instructors could add to our understanding of 
orchestration. As stated earlier, group cognition (Stahl, 2006) assumes that knowledge emerges from 
the interaction of social organization, tool use, and epistemic factors (Goodyear & Carvalho, 2014), 
which are beyond that of the influence of any particular individual. Using custom-built LA tools, such as 
the one suggested in this paper, could give us insight into the underlying mechanisms that support this 
emergence. 

In the introduction, we claimed that theory-led design has the potential to yield innovation in the 
development of LA tools, and that the development of LA tools, and their application, may then 
contribute to learning theory. Whilst there are many available off-the-shelf technologies and 
representations, the argument is that much can be gained by designing domain-specific instruments, 
where the resources and skills to do this are available. By considering stakeholders, objectives, and first- 
principles and having these informed by theory, existing instruments can be adapted and novel 
instruments developed. Whilst clearly such innovation has been occurring and will continue to occur, 
the paper serves to draw attention to the importance of theory-led design. The mindful application of LA 
has the potential to provide us with insights into the activity of learners. By connecting the questions we 
ask of learning theories, we provide a context for the data extracted and analyzed, and potentially 
contribute further to our understanding of that theory, and learning. In this paper, we show that this 
same consideration can be applied to the design of learning analytics tools themselves, and we expand 
the contribution that LA can make, to include our understanding of how learning activity unfolds, and 
how this can be shaped by instructors. 
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